Processing main cause errors and sympathetic errors in devices in a system

ABSTRACT

Provided are a computer program product, system, and method for processing main cause errors and sympathetic errors in devices in a system. Error data for the devices in the system are analyzed to determine a main cause error for one of the devices that cause at least one sympathetic error in the system. A main cause event object for the determined main cause error and at least one sympathetic event object for the determined at least one sympathetic error resulting from the main cause error are generated. A determination is made from the at least one sympathetic event object of at least one sympathetic event action to perform. The determined at least one sympathetic event action is performed to recover from the at least one sympathetic error represented by the at least one sympathetic event object providing the at least one sympathetic event action.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/654,331, filed Oct. 17, 2012, which application is incorporatedherein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a computer program product, system, andmethod for processing main cause errors and sympathetic errors indevices in a system.

2. Description of the Related Art

Devices in a computer system or devices shared by multiple nodes in asystem may generate errors. Errors are detected and processed. A maincause error resulting from a device in a system, such as an adaptor,memory or other component, may cause sympathetic errors in the same ordifferent devices. For instance, a main cause error may comprise a powersource failure and sympathetic errors may comprise the errors in thesystem that result from that power source failure.

Prior art error handling algorithms gather data to analyze the errorsfrom the affected resource to determine the resource that caused theproblem and then set an event against the resource to take recoveryactions against the isolated resource. The prior art error handlingalgorithm may check for detected errors according to an order, such ashigher priority errors first, and then perform a series of if-elsestatements to identify and isolate the errors to determine errorrecovery operations to perform.

SUMMARY

Provided are a computer program product, system, and method forprocessing main cause errors and sympathetic errors in devices in asystem. Error data for the devices in the system are analyzed todetermine a main cause error for one of the devices that cause at leastone sympathetic error in the system. A main cause event object for thedetermined main cause error and at least one sympathetic event objectfor the determined at least one sympathetic error resulting from themain cause error are generated. The at least one sympathetic eventobject indicates a sympathetic event action to perform for thesympathetic error represented by the sympathetic event object. Adetermination is made from the at least one sympathetic event object ofat least one sympathetic event action to perform for the determined atleast one sympathetic event. The determined at least one sympatheticevent action is performed to recover from the at least one sympatheticerror represented by the at least one sympathetic event object providingthe at least one sympathetic event action.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a computing environment.

FIG. 2 illustrates an embodiment of a sympathetic event object for asympathetic error of a main cause error.

FIG. 3 illustrates an embodiment of a main cause event object.

FIG. 4 illustrates an embodiment of operations to generate event objectsfor errors.

FIG. 5 illustrates an embodiment of operations to process a main causeerror transmitted by a system to a remote system.

FIG. 6 illustrates an embodiment of operations to process sympatheticerrors.

FIG. 7 illustrates an implementation of a computer system.

DETAILED DESCRIPTION

Prior art techniques for processing errors may take into account maincause errors and initiate error recovery of the main cause error withouthandling sympathetic errors resulting from the main cause error,especially where the sympathetic error effects a device other than thedevice that produced the main cause error. The error handling withrespect to the main cause error and device producing the main causeerror will not address the sympathetic errors that occur on otherdevices.

Described embodiments provide techniques for generating specific eventobjects for main cause errors and sympathetic errors for devices toallow specific error handling with respect to the main cause error aswell as sympathetic errors. Further, described embodiments allow a maincause error to remain persistent in the system in which the erroroccurred and other systems so that sympathetic errors on any systemresulting from the main cause error can be processed as sympatheticerrors of the main cause error and handled accordingly.

FIG. 1 illustrates an embodiment of systems 2 a, 2 b each having anevent manager 4 a, 4 b to manage events related to errors detected atdevices 6 a, 6 b in the system 2 a, 2 b. The devices 6 a, 6 b maycomprise network adaptors, storage adaptors, storage controllers, memorydevices, storage devices, etc., managed through the system 2 a, 2 b. Theevent manager 4 a, 4 b processes error data 8 a, 8 b comprisinginformation on errors generated at the devices 6 a, 6 b and generatesevent objects 10 a, 10 b representing the errors. An event objectcomprises a code object including information and function calls tohandle the specific error for which it was generated.

The generated event objects 10 a, 10 b include main cause event objects12 a, 12 b for main cause errors resulting form the devices 6 a, 6 b andsympathetic event objects 14 a, 14 b for sympathetic errors resultingfrom one of the identified main cause errors. Sympathetic errors mayoccur at the device having the main cause error or external to thedevice having the main cause error, such as at partner devices orchildren devices or in remote systems from the system including thedevice producing the main cause error, or other devices. The eventobjects 10 a, 10 b further include other event objects 16 a, 16 brepresenting errors that could not be classified as main cause errors orsympathetic errors, which may be processed as stand alone errors.

The systems 2 a, 2 b may communicate over a connection 16, which maycomprise a network or a direct connection, such as a cable, bus or otherinterconnect. Each system 2 a or 2 b may be described as remote withrespect to the other 2 b or 2 a. The systems 2 a, 2 b may comprise nodesin a dual node cluster, such as nodes in a central electronic complex(CEC) that communicate through a bus.

FIG. 2 illustrates an instance of a sympathetic event object 14 _(i)representing one sympathetic error, which may indicate an eventidentifier 50; a sympathetic error 52 for which the sympathetic eventobject 50 is generated; a sympathetic event action 54 to perform, e.g.,cancel or modify, to recover from the sympathetic error 52; event flagsto set 56, such as no log, no recovery, no count increment, etc.; eventflags to reset 58 for the sympathetic error 52; and a sympathetic eventhandler function 60 comprising a pointer to a function called for thisdetected sympathetic error 52 to perform additional actions.

If the sympathetic event action 54 indicates to modify, as opposed tocancel, then actions may be specified in fields 56, 58, and/or 60 toimplement the modify action. A cancel action is indicated if there is noadditional handling for the sympathetic error 52 and that it may becleared without further action in the system 2.

FIG. 3 illustrates an instance of a main cause event object 12 _(i), forone main cause error, including: an event ID 70; a device 72 causing themain cause error; the main cause error 74 represented by the main causeevent object; an active flag 76 indicating whether the main cause errorrepresented by the main cause event object 70 is active orinactive/cleared; an active function pointer 78 to a program todetermine if the main cause error 74 is active, which may check device72 and system 2 a, 2 b settings; local device sympathetic events 80comprising a list of sympathetic event objects 14 _(i) representingsympathetic errors caused or related to the main cause error 74 that areat the device 72 causing the main cause error 74; partner devicesympathetic events 82 comprising a list of sympathetic event objects 14_(i) representing sympathetic errors caused or related to the main causeerror 74 at a partner device of the device 72 causing the main causeerror 74; children device sympathetic events 84 comprising a list ofsympathetic event objects 14 _(i) representing sympathetic errors causedor related to the main cause error 74 at a child device of the device 72causing the main cause error 74; and remote systems 86 (if any) to whichinformation on the main cause error 72 event is transmitted.

A partner device may comprise a device that provides redundancy to thedevice producing the main cause error and a child device may comprise adevice that operates in a relation or under control of the deviceexperiencing the main cause event.

FIGS. 2 and 3 show by way of example certain fields and informationincluded in the sympathetic event objects 14 _(i) and main cause eventobjects 12 _(i). In further embodiments, additional, less, and differentinformation may be included in these objects.

FIG. 4 illustrates an embodiment of operations performed by the eventmanager 4 a, 4 b to generate event objects for detected errors atdevices 6 a, 6 b in the systems 2 a, 2 b. Upon initiating (at block 202)the operation to generate the event objects 10 a, 10 b, the eventmanager 4 a, 4 b may analyze (at block 202) error data 8 a, 8 b for thedevices 6 a, 6 b in the system 2 a, 2 b to determine main cause errorsand sympathetic errors resulting form the main cause errors, as well asto other error types. For each device j having one or more main causeerrors (at blocks 204 through 220), the operations at blocks 206 through218 are performed for each main cause error k for a device j. If (atblock 208) there is no main cause event object 208 for the detected maincause error k in the error data 8 a, 8 b, then the event manager 4 a, 4b generates (at block 210) a main cause event object 12 i indicating themain cause error k in main cause error field 74 and indicating the erroras active 76. In this way, a main cause error remains persistent in thesystem in a main cause event object 12 i. After generating the maincause event object 12 i (from block 210) or if there is already a maincause event object 12 i for a previously detected and processed activemain cause error k, then for each detected sympathetic error in theerror data 8 a, 8 b related to or resulting from the main cause error k,the event manager 4 a, 4 b generates (at block 212) a sympathetic eventobject 14 i identifying the sympathetic error 52 and sympathetic eventactions 54, 56, 58, and 60 to perform to clear the error. Alternatively,the sympathetic event action 54 may indicate to clear the sympatheticerror 52 without further action.

The event manager 4 a, 4 b indicates (at block 214) each sympatheticevent object 14 i for a sympathetic error at the device j having themain cause error k in the local device sympathetic events field 80. Theevents manager 4 a, 4 b indicates (at block 216) each sympathetic eventobject 14 _(i) for a sympathetic error at a partner device of the devicej, having the main cause error k, in the partner device sympatheticevents field 82. The events manager 4 a, 4 b further indicates (at block218) each sympathetic event object 14 _(i) for a sympathetic error at achild device of the device j, having the main cause error k, in thechildren device sympathetic events field 84. In this way, the main causeevent object 12 i indicates related sympathetic event objects forsympathetic errors at the device 6 a, 6 b which caused the main causeerror and at partner and children devices to the device causing the maincause error. This allows for fined grain control and classification ofsympathetic errors in relation to the device 6 a, 6 b that caused themain cause error. Further, certain of the fields 80, 82, or 84 mayindicate no sympathetic event objects if there are none of those type ofobjects.

A main cause error object 12 i persistently indicates whether the maincause error 74 is active, so that if subsequent sympathetic errorsresulting from or related to the main cause error 74 are detected, thosesubsequent sympathetic errors may be indicated in the main cause eventobject 12 i for the main cause error of the sympathetic error. Theindication of whether a main cause error is active may be indicated in afield or parameter external to the man cause event object representingthe main cause error.

FIG. 5 illustrates an embodiment of operations performed by the eventmanager 4 a, 4 b in a remote system 2 a, 2 b that received informationon a main cause error transmitted from another system 2 a, 2 b. Thisoccurs because the main cause error on one device may cause sympatheticerrors on the remote system which are to be addressed. Upon receiving(at block 230) information indicating the main cause error, the eventmanager 4 a, 4 b generates, at the remote system 2 a, 2 b, a main causeevent object 12 _(i) for the transmitted main cause error, comprising aremote main cause error to the remote system 2 a, 2 b. The event manager4 a, 4 b may analyze (at block 234) error data 8 a, 8 b for devices 6 a,6 b at the remote system 2 a, 2 b to determine whether there aresympathetic errors at the devices in the remote system resulting fromthe remote main cause error at the system transmitting the remote maincause error. The event manger 4 a, 4 b at the remote system 2 a, 2 bgenerates (at block 236) a sympathetic event object 14 _(i) for eachdetermined sympathetic error at the devices 6 a, 6 b in the remotesystem 2 a, 2 b, and indicate sympathetic event actions 54, 56, 58and/or 60 in the sympathetic event object 14 _(i).

FIG. 6 illustrates an embodiment of operations performed by the eventmanagers 4 a, 4 b to process sympathetic errors. Upon initiating (atblock 250) the operation, the event manager 4 a, 4 b performs a loop ofoperations at blocks 252 through 274 for each device j of the devices 6a, 6 b. For each main cause event object k for a device j, the eventmanager 4 a, 4 b performs the loop of operations at blocks 254 to 276.The main cause event object k is determined from the event objects 10 a,10 b for the device j (at block 256) and the active function pointer 78indicated in the object k is executed (at block 258). The activefunction pointer 78 checks state information in the system 2 a, 2 b orat least one of the devices 6 a, 6 b in the system to determine whetherthe main cause error represented by the main cause event object k isactive. If (at block 260) the main cause error is not active, then themain cause event object k is indicated (at block 262) as inactive infield 76. If (at block 264) the main cause event object k specifies oneor more remote systems 86, then the event manager 4 a, 4 b transmits (atblock 266) information indicating the inactive main cause error to thedetermined remote system so that remote system 2 a, 2 b may indicate aremote main cause error as inactive so that further sympathetic eventobjects are not generated for sympathetic errors related to the inactiveremote main cause errors. From blocks 264 or 266, control ends becausethe main error is active, which may indicate that all sympathetic errorshave been processed and cleared.

If (from the yes branch of block 260) the main cause error for maincause event object k is active, then the event manager 4 a, 4 bdetermines (at block 268) sympathetic event objects 14 i for the maincause event object k, which may be indicated in the sympathetic eventobject fields 80, 82, 84 of the main cause event object k. The eventmanager 4 a, 4 b performs (at block 270) the sympathetic event actions54, 56, 58, and/or 60 in the determined sympathetic event objects torecover from the sympathetic errors represented by the sympathetic eventobjects. The sympathetic event actions performed may be to only clearthe sympathetic error or modify the error by setting event flagsindicated in field 56, resetting event flags indicated in field 58and/or executing a sympathetic event handler function 60 to performfurther sympathetic event action handling. The event manager 4 a, 4 bmay then clear (at block 272) the sympathetic error indicated in theprocessed sympathetic event object 14 i, such as by removing thesympathetic event object 14 i.

After all the sympathetic errors for a main cause error are cleared,then the main cause error may be indicated as inactive in the main causeevent object k representing the cleared main cause error.

Described embodiments provide techniques for classifying errors as maincause errors and sympathetic errors of the main cause errors, and thengenerating event objects for the main cause errors and the sympatheticerrors to provide specific event code to process the errors so thatsympathetic errors may be specifically addressed and cleared along withthe handling of the main cause error.

The described operations may be implemented as a method, apparatus orcomputer program product using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. Accordingly, aspects of the embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the embodiments may take the form of a computer programproduct embodied in one or more computer readable medium(s) havingcomputer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Reference letters i, j, and k refer to different positive integervalues, and may refer to the same or different number of elements whenused in different contexts.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

The systems 2 a and 2 b may be implemented in one or more computersystems, such as the computer system 302 shown in FIG. 7. Computersystem/server 302 may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.Computer system/server 302 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 7, the computer system/server 302 is shown in the formof a general-purpose computing device. The components of computersystem/server 302 may include, but are not limited to, one or moreprocessors or processing units 304, a system memory 306, and a bus 308that couples various system components including system memory 306 toprocessor 304. Bus 308 represents one or more of any of several types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) bus.

Computer system/server 302 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 302, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 306 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 310 and/or cachememory 312. Computer system/server 302 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 313 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 308 by one or more datamedia interfaces. As will be further depicted and described below,memory 306 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 314, having a set (at least one) of program modules 316,may be stored in memory 306 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. The components of the computer 2 may be implemented asprogram modules 316 which generally carry out the functions and/ormethodologies of embodiments of the invention as described herein. Thecomponents 12, 14, 16, and 18 of the content analytics system 2 may beimplemented in one or more computer systems 302, where if they areimplemented in multiple computer systems 302, then the computer systemsmay communicate over a network.

Computer system/server 302 may also communicate with one or moreexternal devices 318 such as a keyboard, a pointing device, a display320, etc.; one or more devices that enable a user to interact withcomputer system/server 12; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 302 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 322. Still yet, computer system/server 302can communicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 324. As depicted, network adapter 324communicates with the other components of computer system/server 302 viabus 308. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 302. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

Further, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may be configured to work in alternate orders. In otherwords, any sequence or order of steps that may be described does notnecessarily indicate a requirement that the steps be performed in thatorder. The steps of processes described herein may be performed in anyorder practical. Further, some steps may be performed simultaneously.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The illustrated operations of the figures show certain events occurringin a certain order. In alternative embodiments, certain operations maybe performed in a different order, modified or removed. Moreover, stepsmay be added to the above described logic and still conform to thedescribed embodiments. Further, operations described herein may occursequentially or certain operations may be processed in parallel. Yetfurther, operations may be performed by a single processing unit or bydistributed processing units.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

What is claimed is:
 1. A method, comprising: analyzing error data fordevices in a system to determine a detected main cause error for one ofthe devices that cases at least one sympathetic error in the system;generating a first main cause event object for the main cause error;generating at least one sympathetic event object for the determined atleast one sympathetic error resulting from the main cause error, whereinthe at least one sympathetic event object indicates a sympathetic eventaction to perform for the sympathetic error represented by thesympathetic event object; determining from the at least one sympatheticevent object at least one sympathetic event action to perform for the atleast one sympathetic error; performing the determined at least onesympathetic event action to recover from the at least one sympatheticerror represented by the at least one sympathetic event object providingthe at least one sympathetic event action, wherein the first main causeand sympathetic event objects are generated for the detected main causeerror at the device in order to perform recovery operations to addressthe detected main cause error; transmitting information indicating themain cause error to a remote system having devices; generating, by theremote system, a second main cause event object for the transmitted maincause error, the second main cause event object comprising a remote maincause error; and analyzing, by the remote system, error data for thedevices at the remote system to determine whether there is at least onesympathetic error at the devices in the remote system resulting from theremote main cause error.
 2. The method of claim 1, further comprising:using the first main cause event object to determine whether the maincause error represented by the first main cause event object is active,wherein the determining and the performing of the at least onesympathetic event action are performed in response to determining thatthe main cause error is active.
 3. The method of claim 2, wherein theusing the first main cause event object to determine whether the maincause error is active comprises determining from the first main causeevent object for the main cause error an active function pointer; andexecuting the active function pointer to check state information in thesystem or at least one of the devices in the system to determine whetherthe main cause error is active.
 4. The method of claim 1, furthercomprising: indicating the main cause error as active in persistentinformation in the system; and generating sympathetic event objects forsympathetic errors resulting from the main cause error indicated in thepersistent information as active.
 5. The method of claim 1, furthercomprising: generating at least one sympathetic event object for thedetermined at least one sympathetic error at the devices in the remotesystem indicating a sympathetic event action; determining, at the remotesystem, at least one sympathetic event action to perform for the atleast one sympathetic error in the remote system resulting from theremote main cause error; and performing the determined at least onesympathetic event action at the remote system.
 6. The method of claim 5,further comprising: receiving, by the remote system, indication that theremote main cause error is inactive; and indicating, by the remotesystem, the remote main cause event error as inactive.
 7. The method ofclaim 1, wherein the first main cause event object indicates at leastone sympathetic event object for at least one sympathetic error at thedevice causing the main cause error.
 8. The method of claim 7, whereinthe first main cause event object indicates at least one sympatheticevent object for at least one sympathetic error for at least one devicethat does not cause the main cause error.
 9. The method of claim 7,wherein at least one device that does not cause the main cause errorhaving the at least one sympathetic error comprises at least one of aredundant partner device providing redundancy for the device having themain cause error and a device comprising a child device to the devicehaving the main cause error.
 10. The method of claim 1, wherein the atleast one sympathetic event actions comprise either a cancel of the atleast one sympathetic event object or performing operations in thesystem to recover from the at least one sympathetic error represented bythe at least one sympathetic event object.
 11. The method of claim 10,wherein the operations performed in the system for the at least onesympathetic event action comprises at least one of setting system flags,resetting system flags, and calling a sympathetic event handlerfunction.
 12. The method of claim 1, further comprising: indicating thefirst main cause event object as inactive in response to determiningthat system and device state information indicate that the main causeerror represented by the first main cause event object is not present.